TIKA-4777: Expose presentation start of delayed QuickTime timed metadata tracks#2936
TIKA-4777: Expose presentation start of delayed QuickTime timed metadata tracks#2936dschmidt wants to merge 5 commits into
Conversation
…data tracks QuickTime timed metadata tracks (mebx sample descriptions, ISO 14496-12 boxed metadata) declare their key names in the file, and the track's presentation start is defined by its edit list: a leading empty edit delays the media by its duration. Parse the mebx key declarations (keys/keyd) and emit, for every declared key of a delayed track, the presentation start in microseconds as <key name>.track-start-us. Undelayed tracks (start 0) are not reported, their start carries no information. This is deliberately generic, no key names are hardcoded. The main use case are Apple Live Photos, which mark the moment the paired still image was captured as the single one-tick sample of the still-image-time track, shifted to that moment by an empty edit. The sample value itself is a constant -1 marker, so the moov boxes alone are sufficient and mdat is never touched. The handler stays active for timed metadata tracks (handler type 'meta') instead of switching to the extraction-free Mp4MetaHandler, which would keep it from seeing the track's stsd. Verified against a real Live Photo video (iPhone 15 Pro, iOS 18.5): com.apple.quicktime.still-image-time.track-start-us = 1233333 (1.2333s, matching the empty edit of 740/600s), likewise the transform keys of the same track; the undelayed video-orientation and live-photo-info tracks are not reported.
9ae502f to
3820b30
Compare
|
Feedback from claude: |
…ist handling Architecture: instead of suppressing the base handler's Mp4MetaHandler and accepting minf/stbl/stsd globally (which also slurped the sample descriptions of unrelated track types), return a TikaMp4MetaHandler subclass from the hdlr branch. The base Mp4MediaHandler already routes the track's stsd and stts; the subclass fills in the empty processSampleDescription/processTimeToSample overrides. This also keeps Mp4MetaDirectory alive. The redundant processMovieTimescale is replaced by the movie timescale the base handler already stores (Mp4Directory.TAG_TIME_SCALE). Edit list correctness: only leading empty edits delay a track, so the scan stops at the first normal entry and sums consecutive leading empty edits instead of picking the first -1 entry anywhere. The per-track state is now reset when the trak container starts rather than at tkhd, making it independent of box order, and the microsecond conversion is guarded against overflow. Sample count invariant: the track start is only the time of the still sample because the track has exactly one sample, so stts is now read and tracks with more than one sample are not reported (previously the delay filter only correlated with this). Tests: the crafted fixture now has realistic track plumbing (gmhd, dhlr/alis data handler, dinf/dref, full stsd/stts/stsc/stsz/stco) and three additional timed metadata tracks covering a version-1 edit list, a non-leading empty edit directly after the delayed track (state leak probe), and a delayed multi-sample track. Emission uses add() instead of set() so distinct tracks declaring the same key do not silently overwrite.
|
Thanks, that is remarkably thorough feedback, and almost all of it was right.
Two shape questions I deliberately did not touch and would like your call
|
|
How about this? from my agent... |
|
My agent wants to know: |
|
LOL.. dedicated new property. I prefer "qt:" as the namespace, but I defer to fellow humans in the conversation. |
|
I feel qt: is a bit unspecific? How about quicktime: or video:qt:? |
|
Went with |
…mage-time Review outcome: instead of grafting a Tika-invented .track-start-us suffix onto Apple's key namespace, the presentation start of the still-image-time track is now a single dedicated property, quicktime:still-image-time (microseconds). It is emitted for any single-sample mebx track declaring com.apple.quicktime.still-image-time, including 0 when the track has no leading empty edit, so "still is the first frame" (0) stays distinguishable from "not a Live Photo" (absent). Foreign mebx keys no longer produce any output. The quicktime: namespace follows the naming used by ExifTool, exiv2 and metadata-extractor, which all spell out QuickTime; the keys are container-level and appear identically in audio-only files, so a video: prefix would be misleading. The unit stays out of the property name per the convention of xmpDM:duration, geo:alt and audio:bitrate. New fixture testMP4_StillImageTimeZero.mov covers the undelayed case (version 1 edit list containing only a media edit).
|
Reworked as discussed: the per-key |
|
From my agent: |
… reset A crafted sample description declaring an entry size like 0xFFFFFFFF passed the < 16 guard, turned negative in the int cast and escaped parse() as a NegativeArraySizeException. Entry sizes beyond the remaining payload are now treated as malformed; the regression test reproduces the escape when the guard is removed. The still-image-time zero fixture now places a delayed foreign-key track before the undelayed track, so an empty edit duration leaking between tracks would surface as a non-zero value in the existing assertion. Also collapses leftover blank lines in TikaMp4BoxHandler.
|
Good catch on the entry size, that one was real: without the guard the crafted stsd escapes parse() as a NegativeArraySizeException (the new regression test reproduces exactly that when the guard is removed). Entry sizes beyond the remaining payload are now rejected as malformed. I also took the test gap point: the zero fixture now places a delayed foreign-key track before the undelayed still-image-time track, so an empty edit duration leaking between tracks would surface as a non-zero value in the existing assertion. Blank lines are gone too. The moov-level hdlr point I left alone: it is latent as you say, real-world container-level metadata declares 'mdta' there rather than 'meta', so guarding against it felt speculative. |
JIRA: https://issues.apache.org/jira/browse/TIKA-4777
Builds on the QuickTime metadata support from #2935 (TIKA-2861, now merged).
QuickTime timed metadata tracks (
mebxsample descriptions, ISO 14496-12 boxed metadata) declare their key names in the file, and a track's presentation start is defined by its edit list: a leading empty edit delays the media by its duration. Following review, this now emits a single dedicated property instead of the earlier per-key suffix scheme:quicktime:still-image-time, the presentation start in microseconds of a single-sample mebx track declaringcom.apple.quicktime.still-image-time. An undelayed track yields 0, so "the still is the first frame" (0) stays distinguishable from "not a Live Photo" (absent). Other mebx keys produce no output.Apple Live Photo videos mark the moment the paired still image was captured as the single one-tick sample of that track, shifted there by an empty edit. The sample value itself is a constant -1 marker, so the moov boxes alone are sufficient and mdat is never read. Apple's own LivePhotosKit JS player derives its
photoTimethe same way (it parses trak/elst/mebx client-side).The
quicktime:namespace follows ExifTool, exiv2 and metadata-extractor, which all spell out QuickTime for this group; the keys are container-level and appear identically in audio-only files, so avideo:prefix would be misleading. The unit stays out of the property name, matchingxmpDM:duration,geo:altandaudio:bitrate.Verified against a real Live Photo video (iPhone 15 Pro, iOS 18.5):
quicktime:still-image-time = 1233333(1.2333 s, matching the empty edit of 740/600 s); the undelayed video-orientation and live-photo-info tracks produce nothing.Testing:
testMP4_QuickTimeMetadata.movasserts the emitted value plus the negative cases (foreign keys, non-leading empty edit, multi-sample track); the newtestMP4_StillImageTimeZero.movcovers the declared-but-undelayed track (version 1 edit list with only a media edit) yielding 0.